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ABSTRACT 


The problem addressed in this research is the need for supervisory or system 
summary displays for the Advanced Tomahawk Weapons Control System (ATWCS). 
These displays are needed to accurately depict the current system state and weapon status 
in order to aid strike supervisory personnel in making correct and timely decisions. This 
research examined the problem in the context of designing a set of graphical displays that 
extracts information relevant to the strike supervisor from ATWCS and displays it in a 
manner that allows both rapid and accurate interpretation. 

The approach used to solve the problem progressed in four distinct phases. The 
first phase, Requirements Analysis, consisted of gathering system requirements through 
interviews with U S. Navy officers who have experience as strike warfare supervisors. In 
the second phase, an initial design was produced using Century Computing’s rapid 
prototyping tool TAE Plus Workbench™. The third phase involved the heuristic and 
guideline evaluation of the prototype based on accepted user interface design principles 
and ATWCS user interface requirement specifications. This evaluation produced a second 
iteration prototype that was used in the final phase. Usability Testing. The prototype was 
tested by U S. Navy Officers with Tomahawk strike experience and test results were 
recorded. Changes were then made to the prototype to correct usability problems 
discovered by the user testing, yielding a third iteration prototype. 

The final result of this research is a set of prototype displays, in both paper and 
TAE Plus Workbench™ resource file formats, that will be provided to Naval Command, 
Control, and Ocean Surveillance Center (NCCOSC) Research, Development, Test and 
Evaluation Division (NRaD) for consideration during system design and implementation. 
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1. INTRODUCTION 


A. BACKGROUND 

Of the 124 surface combatant ships currently active in the U S. Navy, 63 are 
equipped to launch Tomahawk cruise missiles (Sharp, 1995). Of the ten ships that are 
planned for construction through Fiscal Year 1996, eight will be equipped to launch 
Tomahawk cruise missiles (Sharp, 1995). From these statistics one can see that 
Tomahawk cruise missiles do and will play a large role in the U S, Navy’s operations. 

Because Tomahawk missiles have the ability to strike a wide variety of targets 
hundreds of miles inland, they provide operational commanders with many options. 
Tomahawk cruise missiles can destroy targets that jeopardize flight crews, such as 
surface-to-air missile sites. During contingency operations, they provide a rapid reaction 
capability in response to a quick change in hostilities. Tomahawk is a dependable weapon 
that can be used to strike a wide range of targets giving operational commanders a 
versatile tool with which to conduct wartime operations. 

Tomahawk land attack cruise missiles (TLAMs) are complex missiles with equally 
complex firing procedures. Because of this complexity, strike watch teams typically have 
five members: A Database Manager (DBM), Engagement Planner (EP), two Launch 
Controllers (LCs) and an Engagement Control Officer (ECO), also called the Surface 
Strike Warfare Coordinator (SSTWC) on some ships. All watch team members, except 
the ECO, are seated at an Advanced Tomahawk Weapon Control System (ATWCS) 
console at which they carry out the tasks associated with their position, ATWCS is the 
shipboard system that encapsulates all the control functions necessary to launch TLAMs. 

The ECO, acting as the overall strike supervisor, provides guidance to each of the 
strike team watchstanders based on current mission requirements and the tactical situation. 
The ECO also advises the Commanding Officer (CO), Tactical Action Officer (TAO) and 
Officer of the Deck (OOD) on all matters concerning a TLAM strike. The ECO’s duties 
include keeping the CO and TAO informed of the mission status. This consists of 
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information such as any new TLAM mission tasking received, the progress of TLAM 
missile warm-up, the current operational status of the system, etc. The ECO also keeps 
the OOD informed of launch positions and launch times. 

To effectively perform his duties and make proper decisions, the ECO needs 
access to a variety of information available from the weapon control system. Currently, 
there is no single resource in ATWCS that provides this information. Instead, the 
information is scattered throughout the various subsystems; database management, 
engagement planning and launch control. As a result, the ECO must frequently interrupt 
console operator’s tasks to receive updates to time critical information. 

To further complicate matters. Tomahawk mission tasking is usually quite intense, 
requiring the ECO to track multiple engagements, each with multiple missiles, 
simultaneously. This fact is supported by historical data from Operation Desert Storm, 
the first use of Tomahawk cruise missiles in combat (Office of the CNO, 1991). Cohen 
(1993) shows that during the first day of hostilities, 122 Tomahawk land attack missiles 
(TLAMs) were fired with a steady decline in tasking throughout the first week. In all, 288 
TEAMS were fired from just 16 ships and two submarines with the US S FIFE (DD 991) 
launching 58 missiles during the course of the war (Office of the CNO, 1991). The 
success of TLAM in Desert Storm has made it the weapon of choice for disrupting enemy 
operations and hitting targets when aircraft are either not available or unable to fly due to 
heavy anti-air defenses. More recently, the US S NORMANDY (CG 61) launched 13 
TLAMs into Bosnia on September 10, 1995 to destroy an air defense site (Surface 
Warfare, 1995). Based on this historical data, it is a reasonable assumption that future 
conflicts will see the extensive use of TLAM early in the hostilities. Such tasking will 
require ECOs to keep track of multiple missions whose launches must be timed to the 
second to meet strike objectives. 

The aim of this research is to provide the ECO with a tool to reduce the task load 
associated with supervising a TLAM strike. This will be done by using usability 
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engineering methods to design graphical displays that give the ECO a single source of 
necessary information. These summary displays will be located on a small LCD video 
monitor attached to the top of the standard ATWCS monitor. 

B. OBJECTIVES 

The purpose of this thesis is to present a series of prototype graphical displays for 
ATWCS that are tailored for use by strike team supervisors. This research employs 
usability engineering methods to determine what information strike supervisors typically 
need throughout the course of an engagement and the best way to display it. The product 
of this research is a set of graphical displays that will allow rapid and accurate assimilation 
of system data by strike supervisors. 

There are several objectives to reach before achieving the final goal of a set of 
prototype displays. The first objective is to determine exactly what information the ECO 
needs to perform his duties. This information is analyzed to determine the level of detail 
required by the ECO. Further, the information is prioritized to better determine how it is 
to be presented to the ECO. The second objective is the production of a first iteration 
prototype that not only displays the information accurately, but allows for rapid 
assimilation. This prototype is designed within the established framework of JCM-2154, 
the ATWCS Human-Computer Interface Requirements Specification, and conforms to 
general interface design guidelines. As a fiarther requirement, the displays must fit on the 
size-limited secondary screen. The final, and perhaps most important, objective is that the 
summary displays be simple to use and understand. This objective is met using both 
heuristic evaluation and usability testing with experienced ECOs to reveal usability 
problems and make corrections to the prototype. 

C. INTERFACE ENVIRONMENT 

1. Hardware Environment 

The summary displays, hereafter called ECO displays, presented in this thesis must 
fit on the limited screen space of the ATWCS console’s secondary LCD screen. The 
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overall resolution of the screen is 640 pixels by 480 pixels; however, required interface 
elements reduce the usable screen space to 631 by 395 pixels. The ECO display’s window 
frame further reduces the application area to 611 by 356 pixels. Figure 1 shows the layout 
of the secondary screen along with its associated dimensions. 


Overall window. Including frame. (631 x 395 pixels) 



Figure 1. Secondary Screen Layout. 


JCM-2154 states that supervisory information should appear on the upper display. 
Although the secondary screen’s usable application area is very small, its physical location 
makes it the ideal place for displays intended for ECOs. Due to the nature of the ECO’s 
duties and the lack of a dedicated console for his use, the ECO typically remains standing 
behind the watch team members throughout the strike. The elevated position of the 
secondary screen allows information to be viewed by the ECO without having to lean over 
and interrupt a watchteam member. 

2. Software Environment 

The ECO displays are designed as an addition to an existing system. As such, they 
must strictly adhere to the established HCI guidelines and requirements for the parent 
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system. JCM-2154 is the governing document for all ATWCS HCI requirements. It 
defines both the appearance and the behavior of the interface for ATWCS. 

ATWCS uses the UNIX operating system, X Window System window manager 
and the Open Software Foundation (OSF) Motif graphical user interface widget set. The 
X Window System, or X, provides functions for creating and managing windows as well 
as for drawing in these windows. Motif is a set of preconstructed user interface elements, 
called widgets, that provides a high level application interface with X. Motif provides a 
common look and feel to application programs that use it, providing consistency between 
the interfaces of different applications (Brain, 1992). In other words, a Motif button in 
one application looks and behaves just like a Motif button in another. JCM-2154 further 
defines how each Motif interface element should appear and behave within the framework 
of ATWCS. 

D. INTERFACE DESIGN METHODOLOGY 

The methodology used to design the ECO displays is adapted from the usability 
life cycle presented in Nielsen (1993). Since the displays were designed as an addition to 
an existing system, many steps of the life cycle had already been performed by the parent 
system’s developers, resulting in JCM-2154, a very detailed HCI requirements 
specification. The remaining steps that were needed to design the ECO displays were 
carried out in four phases: requirements analysis, design and prototyping, heuristic and 
guideline evaluation and usability testing. 

The methodology used results in an iterative design process that is focused on the 
needs of the user. The first two phases extract the user’s requirements and transform 
them into an initial prototype designed to conform to existing interface requirements. 
Well-defined sets of heuristics and guidelines ensure the interface conforms with sound 
interface design principles to reveal obvious usability problems. Usability testing subjects 
the prototype to a final evaluation conducted by the user. Problems found in testing are 
corrected to yield a third iteration prototype. 
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E. OUTLINE 

Each of the next four chapters of this thesis present a different phase of the 
methodology. The issues, detailed methods and results of each phase are presented. At 
the end of Chapters III, IV and V, the resulting interface design will be presented. The 
final chapter will include conclusions and recommendations about both the design method 
used and the resulting product. 
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11. REQUIREMENTS ANALYSIS 


The goals of requirements analysis are to define the purpose of a proposed 
software system and determine any constraints on its development (Berzins and Luqi, 
1991). When designing a user interface, requirements analysis also entails determining the 
user tasks that the interface must support, User tasks and needs are defined in terms of 
the users and their environment, with no reference to how the system will support those 
tasks or meet those needs (Lewis and Rieman, 1993). Simply put, requirements analysis 
specifies what a piece of software is supposed to do, not how it is supposed to do it. 

This chapter specifies the requirements for the Engagement Control Officer (ECO) 
displays. These requirements include what information should be on the displays and any 
constraints on the displays. The informational requirements are divided based on a 
taxonomy that considers each piece of information’s importance to the ECO during the 
course of a Tomahawk Land Attack Missile (TEAM) strike. 

A. REQUIREMENTS SOLICITATION 

To best determine what-information would benefit an ECO during a strike, 15 U S. 
Navy Surface Warfare Officers were questioned. Each officer has ECO and Tactical 
Action Officer (TAO) experience from different Tomahawk Land Attack Missile 
(TLAM)-capable platforms. During initial contact, the concept of the ECO displays was 
explained. Because ATWCS is not yet on board most TLAM-capable ships, interviewees 
were given a description of the hardware for the ATWCS console. Each officer was then 
asked to list the information they required during a TEAM strike; particularly, any 
information that requires interrupting a console operator to obtain. Subsequent 
communication involved questioning the interviewees about their replies to further narrow 
their specific requirements. 

All but one interviewee relied on recent experience with TLAM operations to 
determine the information they would find useful. The remaining officer, currently serving 
as Strike Warfare Officer on board a destroyer, conducted several strike simulations in 
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order to determine the information he would find helpfial. During these simulations he 
noted the information that he continually required and that which his chain of command 
requested. 

To ensure the officers not assigned to ships accurately recalled the information 
needed during a TLAM strike, their requirements were compared to those found during 
the simulations. Since the requirements from both sources coincide, those based on 
memory alone are considered accurate. Also, as all of the requirements were listed by at 
least two officers, the requirements are the information required during a TLAM strike. 

B. REQUIREMENT TAXONOMY 

To better decide how to display the information on the ECO displays, the notion 
that one piece of information can be more important than another must be introduced. If 
the informational requirements are not prioritized there is a chance that a critical piece of 
information might not be displayed prominently for the ECO. Likewise, a piece of 
information that is not critical might distract the ECO due to its placement, text size, etc. 
Each requirement is placed into one of three categories having the following 

criteria: 

• Critical - Constantly or frequently updated information that the ECO needs 
repeatedly. Critical information should be visible at all times. 

• Summary - The information in this category, when viewed together, depicts the 
overall system status. Summary information should be displayed together on a 
single screen. 

• Detail - This information provides the details needed by the ECO to perform 
his duties. Detail information can be spread out over several screens but 
should be grouped together with similar information where possible. 

C. REQUIREMENTS 

1. Informational Requirements 

Figure 2 lists the informational requirements resulting from requirements elicitation 
and divides them according to the taxonomy discussed previously. The requirements are 
divided based on comments from the interviewees. 
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CRITICAL 


• Time Until Launch (TUL) of the next missile to be launched 

Course and speed to arrive at launch point of next missile to be launched at its 
Time of Launch (TOL) 

• The status of Officer In Tactical Command Information Exchange System 
(OTCIXS) and the time since data was last received (time late). 

• System mode (Training/Tactical) 

• Overall Vertical Launching System (VLS) status (Fault/No Fault) 

• Bearing, range and ATWCS track number of nearest Critical Contact of 
Interest (CCOI) 

SUMMARY 

• Maximum salvo available of each TLAM variant 

• For the next plan to be launched: 

o Plan number 
o Plan status 
o TUL 

o Time on Target (TOT) 
o TOL 

o Missile alignment status of the last missile 
o Plan Salvo 

o Review Required (Yes/No) 

• Launch Control Unit (LCU) configuration and status 

• Status of Command and Decision (C&D) interface 

• Status of Inertial Navigation System (INS) and its alignment (Forward/Aft) 

• Launchers authorized and selected 

• VLSs authorized 

• VLS controlling Weapon Control System (WCS) 

• VLS mode (Tactical/Simulated) 

• VLS State 

• VLS inventory status 

• VLS Launch Enabled 

• Engagement Planning Console (EPC) mode 

• Launch Control (LC) mode 


Figure 2. Informational Requirements for ECO Displays. 
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I DETAIL 

i 

! 

I • For each active TLAM plan; 

j o Plan Number 

o Plan Status 
o Plan Order 
o Time of Launch (TOL) 
o TUL 

0 Time on Target (TOT) 

o Missile alignment status of last missile 

o Plan Salvo, Ready Spare and Backup (# of missiles) 

o Number of missiles selected 

o Review Required (Yes/No) 

o True and Relative Bearing of missile departure 

o In launch basket? (Yes/No) 

o Bearing, and range to launch point 

o Required course and speed to arrive at launch point at TOL 

• For each missile assigned to a plan: 

o Cell number 
o Missile type 
o Associate plan number 
o Launch Side (Port/Starboard) 
o Alignment status 
o Booster status 
o Missile status 
o tOL 

• For each VLS: 

o Missile type in each cell 
o Plan number of cells/missiles assigned to a plan 
o Any modules with faults 
o Location of any inoperable missiles 
o Location of missiles being aligned 
o Location of missiles that have completed alignment 

• Bearing, range, course and speed of all COIs/CCOIs 


(Figure 2 continued.) Informational Requirements for ECO Displays. 
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2. Interface Requirements and Constraints 

The ECO Displays must conform to the requirements listed in JCM-2154. 
Appendix B of JCM-2154 contains a checklist that can be used to ensure compliance with 
the requirements specification. In addition to the requirements listed in that appendix, the 
interface is constrained by the following physical characteristics: 

• The displays must fit in the secondary LCD screen. 

• The overall window size shall not exceed 631 by 395 pixels, including window 
frame. 

• The application area of the window shall not exceed 611 by 356 pixels. 

• The ECO displays shall use a viewing distance of 26 inches. Based on JCM- 

2154 requirements the font shall be Helvetica with a minimum of 13 point bold 
and a maximum of 24 point with a preferred size of 18 points. 

D. CONCLUSION 

This chapter lists the requirements that fleet ECOs need to conduct a TLAM 
strike. These user requirements drive the design of the ECO displays. The displays must 
incorporate all the requirements in order to meet the needs of the user. The design must 
also take into account the requirement taxonomy and the limited display area of the 
secondary screen. 
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in. INTERFACE DESIGN 


A. DESIGN CONSIDERATIONS 

As with any interface, there are many issues influencing the design of the 

Engagement Control Officer (ECO) displays. When designing a user interface, the 

designer must always keep in mind the function of the interface. The function of the ECO 

displays is to display required information to the ECO during a Tomahawk Land Attack 

Missile (TEAM) strike. This information must be displayed in such a way that the ECO 

can interpret it accurately and quickly. 

If the system does not present the information that the user needs, or 
presents it in an unusable or confusing manner, the user may decide not to 
use the system..., or the user’s ability to perform the necessary task may be 
sharply degraded (Tullis, 1988). 

The small size of the secondary screen is also a primary consideration in the design 
of the ECO displays. All the required information must fit in the limited space allowed on 
the secondary screen. This requires that information be displayed as tersely as possible 
without a loss of information. The information must be both concise and complete to 
prevent inaccurate interpretation by the ECO. 

Mayhew (1992) states “user interface design is a matter of compromise and trade¬ 
off.” Often the goals of accurate and rapid assimilation and minimum screen use seem 
mutually exclusive. In such cases, a compromise must be reached that allows the most 
complete data in the smallest screen space possible. 

The ATWCS Human Computer Interface (HCI) Requirements Specification, JCM- 
2154, clearly defines the framework to which the ECO displays must conform. This 
interface framework must also be considered when designing the ECO displays. 

Although designing an interface under such strict standards may seem restrictive, it 
provides several advantages. Interface standards provide guidelines for the operation and 
visual presentation of interface elements ensuring that a button “looks and feels” the same 
throughout an application. These same interface elements provide standard methods of 
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interaction. (Lewis and Rieman, 1994) For example, to select an option from a list of 
mutually exclusive list is often done using a set of radio buttons (Open Software 
Foundation, 1993). Designing to meet an existing standard allows the designer to 
concentrate on the best way to present the interface rather than the details of how 
interface elements must interact with the user (Lewis and Rieman, 1994). 

B. PROTOTYPING 

According to Nielsen (1993), the idea behind prototyping is to quickly and cheaply 
develop something that can be tested with real users. In the case of a user interface, 
prototypes allow interactive user testing to find usability problems before implementation. 
Typically, an interface prototype is tested and modified iteratively as usability problems are 
uncovered and corrected. This strategy is effective because it is normally less expensive 
and time consuming to correct problems during the prototyping phase than after a system 
has been implemented. 

To save cost and time, prototypes are a scaled down versions of the final system, 
lacking either features or functionality of the full system. These two dimensions of 
prototyping are described in Nielsen (1993) as vertical and horizontal prototypes. A 
vertical prototype is one which fully implements only selected features of a system. For 
example, a vertical prototype of an address book program might implement data entry and 
retrieval with real data but no search capability. Vertical prototypes can only be used to 
test portions of a system, although this testing will be under real circumstances with real 
user tasks. A prototype that has a reduced level of functionality is called a horizontal 
prototype. A horizontal prototype is a surface layer that includes the entire user interface 
but has no underlying functionality. (Nielsen, 1993) A horizontal prototype of an address 
book program would include all data entry screens, search dialog boxes, etc., but have no 
ability to store or retrieve data. 

Prototypes can implemented in several ways. An interface design could be 
prototyped by using paper mock-ups. A developer might design the prototype of a fiiture 
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system on a platform different from the final system’s target environment. A prototype 
could also be constructed using a generalized scripting language such as shell scripts 
rather than a true programming language. There also exist many tools designed 
specifically for prototyping. (Nielsen, 1993) Each of these methods have their oAvn 
advantages and disadvantages which are discussed fully in Nielsen (1993). 

The ECO displays are a horizontal prototype constructed with the aid of TAE Plus 
Workbench™, a rapid prototyping tool from Century Computing. TAE Plus 
Workbench™ allows a developer to visually construct an interface and then provide 
limited functionality by using a scripting language. The interface can then be rehearsed in 
real time during development. Another advantage of this tool is that it can generate high 
level language code, such as C or Ada, to implement the interface, shortening 
development time. TAE Plus Workbench™ does have a disadvantage in that it does not 
fully support some functionality found in X-Windows and Motif 

TAE Plus Workbench™ is the development tool for the ECO displays for several 
reasons. First, its visual nature allows it to display the application area and interface 
objects of the ECO displays in their true size. This is especially important because the 
limited space available to the displays must be utilized effectively. Second, TAE Plus 
Workbench™ allows the interface to be rehearsed interactively. This allows users to 
actually see how the ECO displays look and feel during user testing without having to 
write application code. Finally, the tool’s availability and compatibility with JCM-2154 
standards make it an ideal development environment. 

C. OVERALL INTERFACE CONCEPT 

Good interface design is not a matter of applying a set of rules or algorithms to 
achieve a usable interface. As discussed in Mayhew (1992), there are many principles for 
designing a good interface. However, just knowing these principles is not enough to 
ensure a usable interface. A good interface comes from knowing the user, his tasks and 
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his goals. The design of the ECO displays is driven by the nature and the volume of the 
information that it must present. 

Obviously, there are many ways to implement the ECO displays, but one 
characteristic will be shared by any design. Because the amount of information required 
by the ECO is quite large, it is apparent that any design will have to use multiple screens, 
or pages, to fit the information on the secondary screen. This requires a navigation 
method to switch between the display’s pages. 

The nature of the information to be displayed greatly influences the conceptual 
design. Critical information must be displayed at all times implying that some portion of 
the ECO displays be constantly visible. Summary information must be grouped together 
coherently. This suggests that all the summary information be displayed on the same page 
Detail information should be grouped together, indicating the need for pages that contain 
similar information. 

To account for the various categories of information that must be displayed the 
following window layout scheme is used. The basic window of the displays is sized to the 
maximum size for the secondary screen of 631 by 395 pixels, making the application area 
611 by 356 pixels. To provide an area that is constantly visible, the top 45 pixels of the 
application area will remain static. All critical information will appear in this permanent 
status bar. The remaining application area below the status bar will be used for summary 
and detail information. All summary information will appear on a single summary page, 
while detail information will be grouped together and placed on a series of several 
displays. 

As mentioned previously, a navigation method is required to view the various 
pages of the ECO display. Rather than provide a navigation mechanism that would use a 
portion of the application area, such as tabs, the On-Screen Variable Action Buttons 
(OSVs) already present on the secondary screen are used. OSVs are push buttons whose 
changing labels can simulate a menu hierarchy. Their placement adjacent to the ECO 


16 



display’s window and the fact they use no additional application area make them an ideal 
choice. The functionality of OS Vs is fully described in JCM-2154. 

D. DESIGN DETAILS 

As discussed previously, the amount of information to be presented requires the 
ECO displays be divided into several pages. Based on the nature of the informational 
requirements, the displays are split into the following five pages: ECO Summary, 
COI/CCOIs, Forward VLS Status, Aft VLS Status and Plan Status. At the top of every 
page is a status bar that contains all the critical information listed in the informational 
requirements. To navigate between these pages, the OS Vs are configured as shown below 
in Figure 3. 


ECO 

Summary 

Plan 

Status 

VLS 

Status 

CCOI/COIs 


1 0 
V L 


Forward 

Aft 



VLS 

VLS 


Back 


Figure 3. OSV Configuration. 


Because the main objective of the ECO displays is to present information that can 
be interpreted accurately and quickly, particular attention was paid to screen layout. 
There are literally thousands of data-display and screen design principles available to the 
designer, such as those in Schniederman (1992) and Mayhew (1992). Many of these 
guidelines are based on the work of Tullis (1988) who reviewed the results of several 
empirical studies of screen design. Based on the observed results of those studies, Tullis 
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offers many general guidelines on how best to format screen displays for maximum 
accuracy in minimum time and space. Throughout the design of the ECO displays, the 
principles in Tullis (1988) are to be applied whenever possible. 

E. DESIGN DECISIONS 

1. Layout of Critical Information on the Status Bar 
CL Alternative Approaches 

The critical information could be placed on the status bar in just about any 
arrangement. The limited vertical size of the status bar (45 pixels) prevents grouping 
information in vertical columns over 2 rows high. 
b. Solution and Discussion 

Tullis (1988) presents several options for the sequence of data 
presentation. The most applicable of these in the case of the critical information is 
sequencing by importance. Another consideration, not included in Tullis, is the update 
frequency of the data. 

The critical data can be divided logically as listed below: 

• Next launch data: 
oTUL 

o Course and speed to launch positions 

• OTCIXS information: 
o Status (up or down) 

o Time late of OTCIXS 

• VLS status: 

o Forward VLS status 
o Aft VLS status 

• Training Mode 

• Track number, bearing and range of the nearest CCOI 

One arrangement is to have four columns of two rows each, with the two single element 
groups making a column. Of these, the next launch data information is the most time 
critical, therefore it should be the first column. The OTCIXS data can be considered least 
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critical in that the loss of the OTCIXS link will not prevent a missile launch, consequently, 
it should be in the last column. Of the two remaining groups, neither one has any reason 
to precede the other, so any arrangement of these two is acceptable. 

2. Navigation Method Between Pages 

a. Alternative Approaches 

There are several navigation methods that would be effective for the ECO 
displays. These include tabbed pages, push buttons attached to the status bar, a menu bar, 
a pop-up menu, hot-keys and the OSVs. 

b. Solution and Discussion 

Of all the alternatives mentioned above only the last three do not require 
application area be taken away from the ECO displays. Because space is so limited, the 
first two alternatives are not viable options. 

The OSVs are the best of the remaining three options for the following 

reasons: 


• Hot keys, while they normally offer short cuts for experts, require a two handed 
operation such as Control-K. The OSVs, for all but the VLS status pages, 
require only a single keystroke. For the VLS status pages, two keystrokes are 
necessary. 

• A pop-up menu requires mouse interaction within the application area of the 
window. Because the ECO displays are on the secondary screen, mouse 
interaction can be distracting to the console operator. One of the purposes for 
the ECO displays is to prevent interruption of the console operator’s tasks, 
therefore a pop-up menu is not the best choice. If mouse interaction is desired, 
the OSVs can be pushed just like any other pushbutton. 

• The OSVs’ proximity to the ECO displays connect the two visually. 

3. Method of Grouping Information 
0 . Alternative Approaches 

To display many items on the same screen, it is helpful to organize them 
into semantically related groups (Mayhew, 1992). Because space is so limited on each 
page of the ECO displays, this is especially critical. To set off the groups from one 
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another there are three options presented in Tullis (1988): white space, graphical 
boundaries and highlighting, 

b. Solution and Discussion 

Because the font used in the ECO displays is Helvetica 14 point bold as 
required by JCM-2154, highlighting becomes a difficult issue. To highlight an entire 
group would mean that quite a bit of information would have to italicized, displayed in 
reverse video or have the intensity changed. However, any of these techniques would only 
serve to clutter the display. Because of the density of information displayed on the various 
pages, white space alone is not sufficient to convey the visual concept of information 
groups. Graphical boundaries are the best option for setting apart groups of information 
in the ECO displays. They enable the information to be set apart without cluttering up the 
display or using excessive space. Although any graphical boundary might be effective, a 
logical choice from the programmers point of view is the X-Workspace. 

Labeling of each group should be obvious and easily associated with the group. 

A label in a large font (24 point bold) placed near the group is sufficient. 

4. OSV Hierarchy 

a. Alternative Approaches 

The four OSVs must allow the user to access five pages: ECO Summary, 
COI/CCOI, Forward VLS Status, Aft VLS Status and Plan/Missile Status. The changing 
labels on the OSVs allow almost any possible layout. Two possible alternatives include: 

• The four OSVs are labeled to access the four pages not currently displayed. 

When an OSV is pressed its label changes to the last page displayed. 

• Have OSVs labeled for ECO Summary, COI/CCOI and Plan/Missile Status. The 
final OSV is labeled VLS Status. Upon depressing VLS Status all buttons are 
cleared and two buttons are relabeled for Forward and Aft VLS. The display is 
changed after the user selects their desired launcher. 

b. Solution and Discussion 

The first option discussed above is not an acceptable solution because it 
violates the idea of consistency. Each time an OSV is pressed the label changes to the 


20 




previous page displayed. This means that the order of the OSVs will change constantly. 
This prevents the user from remember which keys correspond to which page. 

The second option is more consistent as all the first level button labels 
always appear in the same positions. However, it does not provide any mechanism to 
back out of the second level labels if the user decides not to look at a VLS launcher or hit 
the incorrect key. 

A better option is to extend the second level of the second option discussed 
above. The addition of a button to return to the previous level enables the user to change 
his mind or back out in the case of a incorrect key press. 

5. Method of Highlighting Information 

a. Alternative Approaches 

There are many methods to highlight information including reverse video, 
color, brightness or boldness, underlining and flashing (Tullis, 1988). Another method to 
highlight a piece of information is to increase its size relative to other information on the 
display. 

b. Solution and Discussion 

The font size requirements listed in JCM-2154 for a 26 inch viewing 
distance indicate that a minimum font of 13 point bold is required. To enhance readability 
at this distance, all text is bold. This eliminates the use of this mechanism to highlight 
information. 

Due to the information density that is necessary to place all required 
information on the ECO displays, highlighting methods such as highlighting, increased 
brightness, small font size changes and changing font color will likely not be very effective. 
For a highlighting method to be effective on a very densely packed display, it must truly 
set the information apart from the rest of the display. This is best done with a large font 
change or with a combination of reverse video and color. “Two to four different character 
types used in consistent ways are optimal.” (Mayhew, 1992) Also, evidence shows that 
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reverse video is a very powerful search que and is useful to indicate that an item has been 
selected or to indicate an error field (Mayhew, 1992). 

6. Labeling Conventions 

a. Alternative Approaches 

There are many ways to label information on the screen. Labels could be 
placed above or to one side of the data. The labels could be all capital letters or 
highlighted in some way. However the labeling is to be accomplished, “every data item on 
a screen should be labelled in some way.” (Tullis, 1988) 

b. Solution and Discussion 

Labels in the ECO display will account for a lot of the screen space used, 
therefore some consistent method should be implemented. Such a scheme follows: 

• Labels should contain both upper and lower case letters. Mixed case text is 
easier to read than all uppercase text. Studies show a 13% increase in reading 
speed for mixed case text. (Mayhew, 1992) 

• If many labels appear in a column, they should be right justified. Left justified 
labels can lead to too much space between labels and data elements. (Tullis, 

1988) 

• Each label should be followed by a colon. This provides consistency among 
labels and identifies text as a label and not a data element. 

• Labels should reflect correct user terminology. Tullis (1988) states that, based 
on empirical data, familiar data formats, concise wording and abbreviations 
should be used. More commonly, this is known as speaking the user’s language 
(Nielsen, 1993). 

There might be times when such a labelling scheme cannot be followed. 

This is a definite problem when dealing with large amounts of data in extremely small 
screen space, such as in the ECO displays. In such cases, as many of the above guidelines 
as possible should be used. 
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7. Displaying Information That Will Prevent Launch 

a. Alternative Approaches 

Any data that prevents launching a missile should be brought to the 
attention of the ECO. This requires that the information be highlighted in such a way that 
it is easily picked out from other data on the display. As discussed above, options include 
reverse video and color or increased font size. 

b. Solution and Discussion 

Increased font size is not a good choice for highlighting information in this 
case. Because of the density of the information, labels and information must be spaced 
close together. Increasing the font size of a piece of information may cause it to interfere 
with nearby data. 

“Color is very effective in drawing attention.” (Mayhew, 1992) It can also 
be used to convey status to help the user determine the nature of the message even before 
reading it (Mayhew, 1992). Although simply changing the color of the text would be 
effective, redundant coding enhances performance (Mayhew, 1992). Thus, reverse video 
should be used to provide redundant coding. A combination of reverse video and color is 
a good choice to highlight information that will prevent launch because it provides a 
redundant way to draw attention to this important information. 

8. Displaying Missile Progress 

a. Alternative Approaches 

There are two basic options when deciding how to display missile 
alignment status: textual or graphical. The process proceeds in eight modes, each having 
their own name. Previous versions of TWCS used a textual representation that listed the 
missile’s alignment mode. 

b. Solution and Discussion 

Tullis (1988) showed that initially, graphical representations of simple 
systems, such as the missile alignment status, allowed for faster interpretation that 
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narrative formats. However, as users became more used to both types of display, there 
was no difference in interpretation times. As the ECO displays should be usable at first 
glance a graphical display is the better choice. Also, a progress bar is widely becoming the 
standard metaphor for system task completion, an appropriate widget for displaying 
missile alignment progress. 

9. VLS Layout 

0 . Alternative Approaches 

A full size VLS launcher is has eight modules of eight cells each. 

Displaying the information for 64 launcher cells in a very limited display area does not 
allow for many alternatives. Any method that is chosen should accurately represent the 
physical layout of the VLS launcher. 

b. Solution and Discussion 

As the ECO displays are being added to an existing system, the best option 
is to display the VLS as it is in the parent system. However, the limited application area 
of the ECO displays will require slight modification to ensure the display fits in the display. 
The only significant change to make is to simply reduce the amount of whitespace in the 
display. 

10. Representing Information About Cells and Modules With Faults 

a. Alternative Approaches 

The informational requirements dictate what information should be 
contained in the labels for each launcher cell. Displaying if a module or cell has a fault can 
be done by highlighting the cell or module in some way. 

b. Solution and Discussion 

To ensure that this display remains consistent with the rest of the displays, 
the method used to display a cell/module fault should be the same as the method used to 
display conditions that prevent missile launch. The label for any cell or module that 
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prevents a missile launching from that cell or module will be reverse video with a red 
background and white foreground. 

To show that a missile has been selected, a more subtle form of 
highlighting is required to keep the screen from appearing too cluttered. A form of 
graphical highlighting, such as a border around the cell’s information, is one method to do 
this. 

11. Plan Status Page Layout 

a. Alternative Approaches 

The informational requirements list a great deal of information that is 
needed for each active plan. It is obvious that if several plans are active the information 
required to be displayed will quickly fill the limited application area of the ECO display 
window. Therefore, some method of paging or scrolling through the list of active plans is 
required. Two possible methods of accomplishing this are push buttons or scroll bars. 
h. Solution and Discussion 

To display the required data for each plan, several informational groupings 
should be used. As discussed previously, each data item should be labeled. Some form of 
separator should be used between each plan’s status to aid in interpretation. 

The most consistent method to view the large numbers of active plans is to 
use a scroll bar. The data should be displayed within an X workspace and scrolled with 
the associated scroll bar. 

F. INITIAL PROTOTYPE DISPLAYS 

Figures 4 through 8 show the first iteration design of the ECO displays. These 
initial prototypes will undergo heuristic and guideline evaluation to discover any usability 
problems prior to interactive user testing. 


25 









: 


. 4 V 


> 


"P I f '>* 








TUL: HH:lflI:SS | Mode: Fwd ¥LS:ro ^ . OTCIXS: UP 

C/S to LP: BBB/SS T#### : IBBB/DD I VLS: jP.lCi| - Tme ^^te: HH;MM: SS 




Next Launch - Plan #:NN 


Review Required?: HO 
Salvo: HH 

Final Missile: 


fill TUL: ^:MM:SS 
TOL: HH:1*I:SS 
TOT: HH:MM:SS 

Mis son AJJiianac Oxypto 
Data Data Data 


DOT 

Data- 



LCU Status System Status 


‘Mode if; 
State: 


Mak Salvos 

n 'io t i »r» i i i rri ii mu ii> ii n i ‘ i * i i i^^vr"f""n**T*^ i m ■ na i mni ami 

Lac-c: m 
Lac-D: HH 
xLac-cT itH : 
XLaC-D: HH 
^ m MH 

:VLS Status 


> Estab 


~ State: Tactical 

■ I'-'V - • ■ -i Fvd-.;;.~-Aft’< 

Selected" Yes i 


authorized: 




Yes 


Lamich Enabled: Yes ^ 


Figure 4. ECO Summary Display. 






Figure 5. Plan Status Display. 
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Figure 6. COI/CCOI Display. 







Figure 7. Forward VLS Display. 
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Figure 8. Aft VLS Display. 

















IV. HEURISTIC AND GUIDELINE EVALUATION 


The are two general ways to test a user interface: with and without users. Both 
testing with users and testing without users may fail to reveal all the problems with an 
interface. A combination of the two methods, however, significantly improves the chances 
of uncovering ail the usability problems in an interface (Lewis and Rieman, 1994). No 
matter which methods are employed, interface testing is critical to the success of an 
interface. Not only is there a cost savings associated with using interface testing methods, 
but there is increased usability as well (Nielsen, 1993). This increased usability is apparent 
through reduced errors, faster user response times and shorter user training times, all of 
which are essential to weapon systems (Nielsen, 1993). 

Testing an interface with users can be expensive (Nielsen, 1993). It requires 
access to a group of test users that are as representative as possible of the intended users 
of the system (Nielsen, 1993). Sometimes, such users are not readily available or have 
their own time demands. Because of this, testing an interface with users requires a large 
amount of coordination between the testers and users and a great deal of prior preparation 
(Lewis and Rieman, 1994). Also, an interface is not normally ready to be tested by live 
users until it is nearly complete. Changes made to an interface at this stage in the lifecycle 
can be very costly to make (Nielsen, 1993). 

From a time and cost standpoint, testing an interface without test users is relatively 
less inexpensive as it does not require advanced planning or a test user’s time and effort 
(Nielsen and Molich, 1990; Lewis and Rieman, 1994). Testing an interface without users 
can be conducted early in the usability lifecycle to reveal problems that may hinder user 
testing. Problems uncovered at this stage of development are easier to correct as well 
(Nielsen, 1993). 

This chapter discusses three methods of testing an interface without users: 
heuristic evaluation, guideline evaluation and cognitive walkthroughs. The methods are 
compared to other forms of user testing and two methods are applied to the initial 
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Engagement Control Officer (ECO) displays presented in Figures 4 through 8. The results 
of these analyses are listed and the resulting second iteration ECO displays are presented. 

A. ANALYSIS METHODS 

1. Heuristic Evaluation 

Heuristic evaluation is an informal method of usability analysis where evaluators 
use their experience and a set of guidelines, called heuristics, to find usability problems 
with an interface (Nielsen and Molich, 1990). The goal of heuristic evaluation is to find 
usability problems in an interface so they can be resolved as part of the iterative design 
process (Nielsen, 1993). There are many lists of heuristics for interface evaluation. They 
range from very short, very general guidelines to lists containing thousands of very 
specific guidelines. Both approaches have their problems. Evaluations using very general 
lists ofl:en miss usability problems and the long lists are usually too unwieldy to apply 
(Lewis and Rieman, 1994). 

a. A Set of Heuristics 

Nielsen and Molich (1990) introduce a set of nine heuristics for to 
evaluating the usability of an interface. These heuristics were developed by Nielsen and 
Molich based on their experience teaching and consulting about usability engineering. The 
nine principles are generally accepted in the user interface community and are present 
either explicitly or implicitly in most lists of principles for human-computer interface (HCI) 
design (Lewis and Rieman, 1994). Each of the heuristics is discussed briefly below. A 
more thorough discussion of each heuristic can be found in Nielsen (1993). 

• Simple and natural dialog - User interfaces should be as simple as possible, 
presenting only the information the user needs, and no more, exactly when it is 
needed (Nielsen, 1993). This information should be in an order that matches the 
user’s task at hand (Lewis and Rieman, 1994). When graphical user interfaces 
are used, these concepts can be extended to graphical aspects such as color use 
and graphical boundaries. 

• Speak the user’s language - Since an interface is designed for a user, it should 
use terms that are based on the user’s language. When users have their own 
specialized terminology for their domain, the interface should use it, rather than 
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more commonly used everyday language (Nielsen, 1993). System-specific 
engineering terms should be avoided. This concept is further extended to the use 
of metaphors in the interface. Any metaphors that are used should be correct 
mappings from the user’s conceptual model and not present any dual meanings. 

• Minimize user memory load - The user should not be made to remember 
information given by the interface. Information should be left on the screen until 
it is no longer needed (Lewis and Rieman, 1994). Also, the system should be 
built around a small number of rules that apply throughout the interface to make 
it easier to transition from one part to the next without having to relearn or 
remember such rules (Nielsen, 1993). 

• Be consistent - Consistency applies to the way interface elements both appear 
and behave. If a user can apply an action in one situation and get a certain result, 
they should expect similar results when applying that same action in a different 
situation. This behavioral consistency lets users feel more confident when using 
the system as they will know what to expect when repeating the same action. 

The same information should be formatted in the same way on every screen to 
facilitate recognition. (Nielsen, 1993) This consistency in appearance can help 
reduce search times as a user becomes more familiar with an interface and knows 
where to expect certain information. More on consistency can be found in 
Tognazzini (1990). 

• Provide feedback - The system should let the user know how it is reacting to 
their input at all times (Nielsen, 1993). Feedback should be timely and expressed 
in concrete and specific terms. Users should know the effects their actions 

are having on the system (Lewis and Rieman, 1994). 

• Provide clearly marked exits - “The system should offer the user an easy way 
out of as many situations as possible.” (Nielsen, 1993) Exits can be in the form 
of Cancel buttons, escape key sequences or an “undo” facility. However they 
are conveyed to the user, exits should be obvious, easily accessed and quick. 
They should not rely on the user’s memory of an action but rather be labeled and 
always visible. 

• Provide shortcuts - Experienced users should be provided with quick methods 
to accomplish tasks without having to view information they do not need (Lewis 
and Rieman, 1994). They should be able to go directly to a desired 

location in the interface without having to traverse lengthy menu hierarchies. 

• Good error messages - A system should provide error messages that are clearly 
worded, precise, unintimidating and offer solutions (Nielsen, 1993). Systems 
should also provide for error recovery when appropriate. 


33 



• Prevent errors - A better option than having good error messages is to avoid 
error situations in the first place. This is done by recognizing error prone 
situations and either avoiding them entirely or providing a way for the user to 
proceed through the situation without causing an error. An example is to 
provide a list of available files to a user rather than have them type in a filename 
that might be invalid. 

b. How to Apply Heuristics 

Nielsen and Molich (1990) outlines the best way they found to apply 
heuristics to evaluate an interface. They conducted an experiment in which four groups of 
evaluators heuristically evaluated separate interfaces. The average proportions of known 
usability problems found were 51%, 38%, 26% and 20% in the four experiments (Nielsen 
and Molich, 1990). By analyzing the results of their experiment, it was discovered that if 
the usability problems found by individual evaluators were combined with those of other 
evaluators in the group, the overall number of distinct usability problems increased 
dramatically. These aggregates of evaluators can find significantly more usability 
problems as the number of evaluators increases, with a point of diminishing returns around 
the point often evaluators (Nielsen and Molich, 1990). Table 1 shows the average 
proportion of usability problems found in each of the four interfaces for various sized 
aggregates of evaluators. 



Number of Evaluators in Aggregate 

I 

2 

3 

5 

10 

Interface 1 

51% 

71% 

81% 

90% 

97% 

Interface 2 

38% 

52% 

60% 

70% 

83% 

Interface 3 

26% 

41% 

50% 

63% 

78% 

Interface 4 

20% 

33% 

42% 

55% 

71% 


Table 1. Average Proportion of Usability Problems Found for Various Sized Aggregates 
of Evaluators. After Nielsen and Molich (1990). 
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This data indicates that heuristic evaluation is best done with more than 
one person. However, even with only one evaluator, a fairly large number of problems can 
be found. When supplemented with another usability engineering method a large number 
of problems can be discovered with only one evaluator (Nielsen and Molich, 1990). 
Another way to increase the proportion of usability problems found with only one 
evaluator is to use a more experienced evaluator. An even greater number of problems 
can be discovered if the evaluator is also a “double expert,” that is, the evaluator is 
experienced in the interface’s domain as well as in interface design (Nielsen, 1993). 

2. Guideline Evaluation 

Guideline evaluation is a method of interface evaluation that uses very specific 
recommendations about the design of an interface. They usually include such 
recommendations as how the contents of a screen should be organized or how items 
should be arranged in a menu. The number of guidelines is usually very large. Guideline 
lists are normally specific enough to be applied by people unexperienced in user interface 
evaluation (Jeffries et al. 1991). This potentially increases the number of people that are 
able to conduct interface evaluation which is especially helpful when usability specialists 
are unavailable. 

One example of a guideline list can be found in the Open Software Foundation’s 
Motif Style Guide. It includes an 80-page Level One Certification Checklist that specifies 
how an application program must behave to be certified “OSF/Motif Style Guide 
Compliant” (OSF, 1993). Such a list can be invaluable to an interface developer to ensure 
a new interface complies with current standards and to point out possible usability 
problems associated with not following such standards. 

3. Cognitive Walkthrough 

Cognitive walkthroughs are a method of interface evaluation in which interface 
developers walk through the interface in the context of core tasks that a typical user needs 
to accomplish (Jeffries et al. 1991). The evaluation proceeds by first selecting a task that 
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the interface was designed to support. The evaluator then attempts to accomplish the task 
by pretending to be a user who is new to the interface. For the method to work correctly, 
the evaluator must make good assumptions of what the user is thinking. While carrying 
out the task, if the evaluator cannot make a reasonable assumption based on his 
knowledge of the user and the prompts and information supplied by the interface, a 
usability problem exists (Lewis and Rieman, 1994). 

This method is best suited for a task-centered user design methodology such as 
that described in Lewis and Rieman (1994). In such a methodology, the interface is 
designed to support a list of typical tasks which are specified during a task analysis. 
Walkthroughs focus most clearly on problems users will encounter when they first use an 
interface (Lewis and Rieman, 1994). Also, a cognitive walkthrough is really a tool for 
developing the interface, not validating it (Lewis and Rieman, 1994). 

4. A Comparison of the Methods 

Jeffries et al. (1991) experimentally compares four methods of testing an interface: 
heuristic evaluation, guideline evaluation, cognitive walkthroughs and usability testing. A 
single interface was evaluated by four teams, each using one of the above methods. 

Specific information about the background of the interface reviewers and the methods they 
used can be found in Jeffries etal. (1991). 

After the completion of each evaluation, the errors they reported were analyzed. 
The evaluations uncovered 206 problems that addressed usability errors which were 
dubbed “core” problems. Heuristic evaluation uncovered 105 core problems with usability 
guidelines, cognitive walkthroughs and usability testing accounting for 35, 35 and 31 of 
the remaining 101 problems, respectively (Jeffries c/a/., 1991). 

Each core problem was also rated by the testers in terms of severity on a scale of 1 
(trivial) to 9 (critical). These ratings took into account the impact of the problem, the 
frequency it occurred and the number of users that it would affect. (Jeffries et al, 1991) 
Using these ratings, each evaluation method’s mean problem severity was calculated. 
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Usability testing uncovered the most severe problems, on average, with a mean problem 
severity of 4.15. Guidelines, heuristic evaluation and cognitive walkthroughs found 
problems with mean severity of 3.61, 3.59 and 3.44, respectively (Jeffries etal, 1991). 

The experimenters further analyzed the results of the interface evaluations by 
calculating a benefit/cost ratio. The benefit was found by summing the severity scores of 
all the core problems while the cost is simply the number of man-hours spent on analysis. 
The benefit/cost ratio was determined using two methods. In the first, the time that 
evaluators using the guidelines and cognitive walkthrough methods spent becoming 
acquainted with the interface was included. The second ratio does not take into account 
this time because in a real world situation the interface’s developers would be conducting 
these evaluations, therefore, this additional time would not be expended. Table 2 
summarizes the results of the benefit cost analysis using the second method. 



Heuristic 

Evaluation 

Usability 

Testing 

Guidelines 

Cognitive 

Walkthrough 

Sum of 
severity scores 

433 

133 

130 

120 

Analysis time 
(man-hours) 

35 

199 

22 

37 

Severity/time 

12 

1 

6 

3 


Table 2. Benefit/Cost Ratios for the Four Evaluation Techniques. 


After Jeffries eta/., 1991. 


This experiment shows that techniques for testing an interface without users are 
very cost effective, especially heuristic evaluation. However, many severe problems 
simply cannot be found without testing an interface with users (Jeffries etal, 1991) . It 
also implies that a combination of usability testing techniques are the best way to find a 
wide range of usability problems. 
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B. ANALYZING THE INTERFACE WITHOUT USERS 

1. Evaluation Methods Used 

To test the ECO displays without users, two methods are employed: heuristic 
evaluation and guideline evaluation. These methods are used because of their high 
benefit/cost ratio and their ease of application. The combination of these methods can 
uncover a significant number of usability problems as shown by Jeffries et al. (1991). 

The heuristic evaluation of the ECO displays is conducted by one person using the 
heuristics described in Nielsen (1993). Although the optimal way to conduct a heuristic 
evaluation is to use aggregates of evaluators, a single evaluator can uncover a significant 
number of usability problems (Nielsen and Molich, 1990). The results of a heuristic 
evaluation conducted by a single evaluator should not be the sole method for determining 
usability problems because, as shown in Nielsen and Molich (1990), a single evaluator 
can miss some usability problems. In order to uncover the maximum percentage of 
usability problems, single evaluator heuristic evaluation should be only one part of a 
usability testing plan. 

The guidelines used to conduct the guideline evaluation are contained in Appendix 
B of JCM-2154. This 46-page checklist ensures that a user interface conforms to JCM- 
2154 standards. Each guideline represents a single HCI requirement from JCM-2154. 

2. Results of Heuristic Evaluation 

Heuristic evaluation of the ECO displays reveals nine usability problems. The 
small number of problems can be attributed to two reasons. First, the evaluation is 
conducted by only one evaluator and, for the reasons discussed previously, one cannot 
expect this method to uncover a large proportion of usability problems (Nielsen and 
Molich, 1990). Second, the ECO displays are veiy simple in terms of user interaction. 

The displays consist of only five display pages, only three of which have interface elements 
that can interact with the user. Table 3 describes each problem and its resolution. 
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Heuristic 

Problem Description 

Problem Resolution 

ECO Summary 

Speak the user’s 
language 

Several labels are abbreviated 
incorrectly. 

Change labels to more 
acceptable abbreviations. 

EGO Summary 

Be consistent 

Position of information given for 
next launch is not consistent 
with the same data listed on the 
Plan Status Display. 

Change information’s position 
to the same as listed for each 
plan on the Plan Status 

Display 

ECO Summary 

Be consistent 

Information about Salvo is not 
consistent with the information 
given on the Plan Status 

Display. 

Change information given to 
‘‘Salvo/RS/BU” to become 
consistent with Plan Status 
Display. 

ECO Summary 

Be consistent 

Labels of groups are not 
consistent. ‘‘VLS Status” label 
is not centered over the group it 
labelled. 

Move label to center of group. 

ECO Summary 

Be consistent 

Labels of conditions signifying a 
launch is not possible are not 
consistent in their use of 
capitalization. 

Change labels of all 
conditions that prevent launch 
to contain all uppercase letters 
only. 

Plan Status 

Simple and 
natural dialog 

Violates the gesalt rules for 
human perception as described 
in Nielsen (1993). This 
decreases the user’s ability to 
perceive relationships between 
interface elements. 

Modify arrangement of 
information to improve group 
perception. 

Plan Status 

Be consistent 

Method of setting off groups of 
data is inconsistent with that 
used on ECO Summary Display 
and COI/CCOI Display. 

Change grouping method to 
that used on ECO Summary 
Display and COI/CCOI 

Display 

Plan Status 

Be consistent 

! 

Method of labelling groups is 
inconsistent with that used on 

ECO Summary Display and 
COI/CCOI Display. 

Move group labels to a 
position centered above each 
group. 

COI/CCOIs 

Speak the user’s 
language 

Abbreviation for course, “Crs”, 
is not the normal abbreviation 
used. 

Change label to “Cse,” a more 
j standard abbreviation. 


Table 3. Heuristic Evaluation Results. 
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3. Results of Guideline Evaluation 

Using the guideline checklist in JCM-2154, seven guideline violations can be found 
in the ECO displays. Table 4 shows which guidelines are violated and the action required 
to correct the problem. 



JCM-2154 Guideline 

Corrective Action 

COI/CCOIs 

When applications provide the capability to sort 
the contents of a multi-column list box the 
heading of each column shall seem to be a push 
button. 

Remove “Sort by” option menu. 
Replace column labels with push 
buttons whose function is to sort 
the list by that column when 
pressed. 

Forward and Aft 
VLS Status 

A push button shall be used to initiate action. 

Change the representation of each 
cell fi-om a push button to a 
graphic box. 

All 

When a widget contains an error, its background 
shall remain IndianRed2. 

Although this does not apply 
directly, to remain consistent with 
JCM-2154, change background 
color of conditions preventing 
launch to IndianRed2. 

All 

All text shall be of the Helvetica font. 

Change font to Helvetica. 

All 

1 The title shall be centered in the title bar and 
presented in uppercase letters. 

Change titles to uppercase letters. 

Plan Status 

Data groupings shall be indicated with blank 
space, separator lines, and/or different intensity. 

Revise method of grouping 
information. 

COI/CCOIs 

If the window contains many rows and columns, 
a blank line shall be inserted after every third to 
fifth row and three spaces inserted between 
every column. 

Add blank line in between every 
third data set. 


Table 4. Guideline Evaluation Results. 


4. Effectiveness of Heuristic and Guideline Evaluation 

Because one cannot know in advance the number of usability problems an interface 
contains, it is unclear what proportion of the total problems heuristic and guideline 
evaluation uncovers. Further evaluation by additional or more experienced evaluators may 
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reveal a larger number of usability problems (Nielsen and Molich, 1990). Changes made 
to the ECO displays based on the results of the heuristic and guideline evaluation will 
make the interface more consistent with both accepted HCI standards and JCM-2154 
requirements (Lewis and Rieman, 1994). This will reduce chances of user error, increase 
the speed and accuracy of data assimilation and reduce training time for the interface 
(Nielsen, 1993). 

C. SECOND ITERATION DISPLAYS 

Figures 9 through 13 show the second iteration of the ECO displays. These will be 
subjected to user testing to reveal any further usability problems in the interface. 
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Figure 9. Second lieration ECO Summary Display 







Figure 10. Second Iteration Plan Status Display 
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Figure 13. Second Iteration Aft VLS Display. 











































































V. USABILITY TESTING 


“You can’t really tell how good or bad your user interface is going to be without 
getting people to use it.” (Lewis and Rieman, 1994) Testing an interface with users is the 
most fundamental usability testing method because it provides the opportunity to find 
usability problems that can be missed by evaluation methods such as heuristic evaluation, 
cognitive walkthrough and guideline evaluation (Nielsen, 1993), Many of the most severe 
problems in a user interface simply cannot be found without using real users to test the 
interface (Jeffries et a/., 1991). 

This chapter discusses some issues involved in usability testing. One method of 
usability testing, thinking aloud, is explored in detailed and the results of using this 
method on the Engagement Control Officer (ECO) displays is presented. Usability 
problems uncovered by user testing are then corrected and a set of third iteration displays 
is presented. 

A. ISSUES IN USABILITY TESTING 

1. Interface Test Users 

Choosing users to test an interface can be as important as choosing a usability 
method to use. Test users should be as representative of the intended users of the system 
as possible (Nielsen, 1993). If the exact individuals for whom a system is being designed 
can be identified, then they should be used. If such users cannot be identified, subjects 
from closely related user populations might be acceptable subjects. For example, if a 
system is being designed for doctors, medical students may be good test users with more 
available time than practicing physicians (Lewis and Rieman, 1994). 

Another issue when selecting test users is to account for both novice and expert 
users. Because of the disparity in their knowledge, novice and expert users should be 
tested with separate tests. These tests should contain common elements but also include 
different test tasks, (Nielsen, 1993) 
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There are also many ethical concerns when dealing with test users. Users must be 
made to feel comfortable and their emotions and well-being should always be kept in mind 
(Nielsen, 1993), Test users often feel a great deal of pressure to perform. Many times, a 
usability problem with an interface will cause a user to worry that they lack the ability or 
intelligence to use the system correctly. Users should be made to understand that it is the 
system that is being tested, not them. They should also be aware that their inability to 
perform a certain task is not a reflection on their ability, but rather the inability of the 
interface to provide them the means to accomplish it. (Lewis and Rieman, 1994) More 
on the ethical aspects of user testing can be found in Nielsen (1993) and Lewis and 
Rieman (1994). 

2. Types of Data 

Lewis and Rieman classify the data that can be collected during usability testing as 
either “process” data or “bottom-line” data. Process data are observations of what a test 
user is doing during the test step-by-step and, hopefully, why they are doing it. Bottom- 
line data are typically quantitative measurements taken during a usability test, such as the 
time a user takes to complete a task. (Lewis and Rieman, 1994) 

Collection of process data is also know as formative evaluation. The main goal of 
a test using this form of data collection is to determine which aspects of a user interface 
are good or bad and to determine what changes should be made to a design. Formative 
evaluation is often done as part of the iterative design process and can be accomplished 
with only a few users. One usability testing method to use for formative evaluation is the 
thinking aloud test, discussed below. (Nielsen, 1993) 

Summative evaluation is an attempt to assess the quality of an interface by 
collecting and analyzing bottom-line data. This form of evaluation is often done to 
compare two alternative interfaces or as a method to perform a competitive analysis. 
Measurement tests are typically used to perform a summative evaluation. In measurement 
tests, an aspect of the interface is given a quantifiable measurement that will be collected 
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during the usability test. Nielsen (1993) lists several quantifiable usability measurements. 
This bottom-line data is recorded and analyzed to determine how well an interface meets 
expected usability goals or how one interface compares to another. The collection of 
bottom-line data takes a large number of user to ensure the data is both valid and reliable. 
(Nielsen, 1993) 

3. Deciding on a Usability Method 

Choosing a usability method can depend on many factors. But no matter which 
method is chosen, relying on the results of a single usability method to the exclusion of 
others is not recommended. Many usability methods can supplement one another because 
they address different parts of the usability lifecycle and the strengths of one method can 
make up for the weaknesses of another. (Nielsen, 1993) 

Possibly the biggest factor that affects the choice of usability method is the goal of 
the test. One would not choose a qualitative or subjective test, such as thinking aloud, to 
validate a system because the opinions of a few test users may not reflect those of the 
actual target user group. Another factor that is closely related to the test goal is the stage 
of the usability lifecycle in which the test is conducted. Many methods, such as hueristic 
analysis, are best done early in the lifecycle while others, such as performance 
measurements, are more appropriate to the end of the lifecycle. 

Another factor that drives the choice of which usability method to use is the 
number of test users available. When few users are available, it is best to concentrate on 
methods such as hueristic analysis and thinking aloud. Methods such as performance 
measurement require a large number of users in order to gather enough bottom-line data 
for the measurements to be considered valid and reliable. (Nielsen, 1993) 

Table 5 summarizes the advantages and disadvantages of several usability methods. 
The table is necessarily simplified; however, Nielsen (1993) covers all the usability 
methods listed in detail. From Table 5 one can determine which usability method is best 
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suited to a particular situation based on the usability lifecycle stage and number of test 


users available. 


Method 

Name 

Lifecycle Stage 

Users 

Needed 

Main Advantage 

Main Disadvantage 

Hueristic 

evaluation 

Early design, “inner 
cycle” of iterative 
design 

None 

Finds individual 
usability problems. Can 
address expert user 
issues. 

Does not involve real 
users, so does not find 
“surprises” relating to 
their needs. 

Performance 

measurements 

Competitive 
analysis, final 
testing 

At least 

10 

Hard numbers. Results 
easy to compare. 

Does not find individual 
usability problems. 

Thinking aloud 

Iterative design, 

formative 

evaluation 

3-5 

Pinpoints user 
misconceptions. Cheap 
test. 

Unnatural for users. 

Hard for expert users to 
verbalize. 

Observation 

Task analysis, 
follow-up studies 

3 or more 

Ecological validity; 
reveals users’ real tasks. 
Suggests functions and 
features. 

Appointments hard to 
set up. No experimenter 
control. 

Questionnaires 

Task analysis, 
follow-up studies 

At least 

30 

Finds subjective user 
preferences. Easy to 
repeat. 

Pilot work needed (to 
prevent misunderstand¬ 
ings). 

Interviews 

Task analysis 

5 

Flexible, in-depth 
attitude and experience 
probing. 

Time consuming. Hard 
to analyze and compare. 

Focus groups 

Task analysis, user 
involvement 

6-9 per 
group 

Spontaneous reactions 
and group dynamics. 

Hard to analyze. Low 
validity. 

Logging actual 
use 

Final testing, 
follow-up studies 

At least 

20 

Finds highly used (or 
unused) features. Can 
run continuously. 

Analysis programs 
needed for huge mass of 
data. Violations of 
users’ privacy. 

User feedback 

Follow-up studies 

Hundreds 

Tracks changes in user 
requirements and views. 

Special organization 
needed to handle 
replies. 


Table 5. Summary of Usability Methods. From Nielsen (1993). 
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B. THINKING ALOUD 


A combination of usability methods that is often useful is heuristic evaluation and 
thinking aloud. First a hueristic evaluation is conducted to clean up the interface and 
correct any “obvious” usability problems. After redesign, the interface is subjected to user 
testing using the thinking aloud method to check the outcome of the iterative design step 
and to find remaining problems that were not discovered by the hueristic evaluation. 
(Nielsen, 1993) 

There are two major reasons for alternating between heuristic evaluation 
and user testing as suggested here. First, a heuristic evaluation pass can 
eliminate a number of usability problems without the need to “waste users,” 
who sometimes can be difficult to find and schedule in large numbers. 

Second, these two categories of usability assessment methods have been 
shown to find fairly distinct sets if usability problems, meaning that they 
supplement each other rather than leading to repetitive findings. (Nielsen, 

1993) 

'"Thinking aloud may be the single most valuable usability engineering method.” 
(Nielsen, 1993) The idea behind thinking aloud is simple. A user is asked to complete a 
task and, while they work on it, they are to express their thoughts verbally (Lewis and 
Rieman, 1994). This usability method allows testers to understand how the user views an 
interface making it easy to identify any major misconceptions the user might have 
(Nielsen, 1993). The thinking aloud method has the advantage of being able to elicit a 
large amount of qualitative information fi'om a small number of users. Its disadvantage is 
that it does not lend itself to any types of performance measurement. (Nielsen, 1993) 
Lewis and Rieman (1994) describes many of the aspects of the thinking aloud 
procedure. These aspects are summarized in the following list: 

• Instructions - The user should understand that the tester wishes to hear what 
they are thinking about while performing a task. They should understand that 
the interface is being tested, not them. (Lewis and Rieman, 1994) 

• The observer’s role - The observer should remain with the user to provide 
assistance if necessary. Users will not normally give a steady flow of comments 
unless prompted occasionally (Lewis and Rieman, 1994). The observer may 


51 


have to ask questions such as “What do you think that means?” 

when a user becomes confiised to elicit constructive remarks. (Nielsen, 1993) 

• Recording - Information can be recorded by taking notes on paper or may be 
accomplished by videotaping the test user. No matter how it is done, the goal is 
to record the users thoughts while performing an action. 

• Summarizing the data - A list of all the problems the user had should be made. If 
possible, the reason why each difficulty arose should be added to the list. (Lewis 
andRieman, 1994) 

• Using the results - The results should be looked at from two aspects. First, does 
the data show that the interface worked as intended or did the users take 
different approaches than expected? Second, based on the user’s difficulties, 
how important and difficult is each problem to correct? 

C. USER TESTING AND RESULTS 

1. Test Plan 

User testing is carried out using the thinking aloud method. This method is well 
suited to test the ECO displays as they are still in the iterative design portion of the 
lifecycle and few test users are available. Users are instructed to describe their thought 
process while doing so. An observer is present to both record users’ comments and assist 
the users only when necessary. The resulting data is summarized and subsequently used to 
make corrections to the ECO displays. 

2. User Sketch 

Three test users are all U S. Navy officers with experience on board Tomahawk 
Land Attack Missile (TLAM) capable ships. All three users are qualified ECOs and one of 
the users has experience as Strike Warfare Officer, whose primary job is the maintenance 
and employment of the Tomahawk cruise missiles and their support systems. These test 
users are very representative of the users who would use the ECO displays during a 
TLAM strike. 

3. Problems Found and Resolution 

User testing using the thinking aloud method with three test users reveals eight 
additional usability problems. Each problem, the reason why it occurred and its resolution 
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is listed in Table 6. The usability problems found in user testing are the basis for 
corrections to the second iteration prototype displays. 


lEUBlIIIII 

Usability Problem 

Reason for Problem 

Resolution 

All 

The label for the nearest 

CCOI is confusing. Users 
are unable to determine what 
the label or information 
meant. 

The label does not convey 
the proper information. In 
an attempt to save space, the 
label was shortened to be the 
track number of the CCOI. 
Users are unable to make the 
connection. 

Debriefing with users 
found they do not feel the 
information is critical. 

The best solution is to 
remove the label. 

All 

The label for OTCIXS time 
late is misunderstood. 

Improper terminology causes 
misinterpretation by the test 
users. 

Change the label to “Last 
Data:” 

All 

Label for system status 
(Training/Tactical) is not 
obvious enough. 

The label’s position in the 
middle of the status bar 
keeps it from standing out. 

Move system status to the 
left side of the status bar 
to make it more 
prominent. 

Forward 
Vertical 
Launching 
System (VLS) 
Status and Aft 
VLS Status 

Users are unable to 
determine which half 
modules are unavailable due 
to either a missile already 
being selected or a module 
fault. 

Missiles that are unavailable 
for selection are not specified 
in any way. 

Reduce the intensity of 
missiles that are 
unavailable for selection. 

ECO 

Summary and 
Plan Status 

Label for Salvo/RS/BU is 
misunderstood. 

The “BU” portion of the 
label is not needed and 
causes confusion. A “back¬ 
up” mission is an operational 
concept and not something 
specified in the weapon 
system. 

Remove “/BU” from 
label. 

ECO 

Summary 

Information for Controlling 
Weapon Control System 
(WCS) causes confusion. 

The information does not 
represent the true state of the 
WCS switch. 

Change information for 
Controlling WCS to read 
Off, Manual or Auto. 

ECO 

Summary 

Label for Inertial Navigation 
System (INS) causes 
confusion. 

The INS is normally referred 
to by its equipment 
designator WSN. 

Change label to read 

WSN instead of INS. 


Table 6. Usability Testing Results. 
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Display 

Usability Problem 

Reason for Problem 

Resolution 

OSV Hierarchy 

To change the page to 
one VLS status page 
while viewing the other 
requires too many 
keystrokes. Users 
prefer to switch back 
and forth between these 
two pages easily. 

After selecting a VLS 
status page, the OS Vs 
return to their original 
state. To switch to the 
other VLS status page 
requires selecting the 
‘VLS Status’ button 
again and then pressing 
the OSV for the desired 
VLS status page. 

When the user selects a 
VLS status page for 
display, the VLS Status 
OSV should be 
relabeled to the other 

VLS status page. For 
example, when viewing 
the Forward VLS Status 
page, the OSV would 
read “Aft VLS Status.” 
This revised hierarchy 
enables fast switching 
between VLS status 
pages. 


(Table 6 continued.) Usability Testing Results. 


D. THIRD ITERATION DISPLAYS 

The displays in Figures 14-18 are the third iteration of the ECO displays. These 
prototype displays can be used to conduct a summative evaluation to determine the 
effectiveness of the interface design. 
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Figure 14. Third Iteration ECO Summary Display 
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Figure 15. Third Iteration Plan Status Display. 






Figure 16. Third Iteration COI/CCOI Display. 
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Figure 17. Third Iteration Forward VLS Display. 

























































Figure 18. Third Iteration Aft VLS Display. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

Iterative interface design is an excellent method to produce a user interface that is 
focused on the needs of the user. Throughout the usability lifecycle, the interface design 
should be evaluated using an appropriate usability method. This ensures that usability 
problems are found as early as possible in the design process. This, in turn, enables 
usability problems to be corrected when they are easiest, and cheapest, to fix. 

Assessing the usability of an interface does not necessarily require large numbers 
of test users or expensive usability testing laboratories. Methods such as heuristic and 
guideline evaluation provide a means to effectively evaluate an interface without the need 
for any users. They also have the advantage of requiring little preparation, making them 
easy to apply. The main disadvantage of evaluating an interface without users is that they 
are simply unable to replace the insights that a real user can provide. An excellent method 
of testing with real users is thinking aloud. This method of user testing reveals what a 
user is think about while using an interface. These insights are invaluable for correcting 
user misconceptions. 

The Engagement Control Officer (ECO) displays are the result of applying an 
iterative design process. By employing usability methods early in the iterative process, 
usability problems in the displays are corrected before they become costly to fix. Testing 
the ECO displays with test user that are representative of the intended user group ensures 
that the displays meet user requirements and agree with user perceptions. 

The ECO displays are a valuable addition to the Advanced Tomahawk Weapon 
Control System (ATWCS). The information that they present is necessary for the ECO to 
complete the mission of launching a successful Tomahawk Land Attack Missile (TEAM) 
strike. The system state and weapon status are accurately depicted to aid the ECO in 
making correct and timely decisions. By using sound screen design principles, the displays 
present the information in such a way that it can be read accurately and rapidly by the 
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ECO. Usability testing, both with and without test users, shows that the displays are 
consistent, use proper terminology and do not present information in a way that it could 
easily be misinterpreted. 

B. RECOMMENDATIONS 

The following list of recommendations are suggestions to improve the usability of 
the ECO displays. They also include recommendations to improve the usability of 
ATWCS for ECOs. 

• Perform additional requirements analysis after the installation of ATWCS in 
fleet units - User perceptions and expectations may change after they become 
familiar with the capabilities of the new system. Additional requirements analysis 
will ensure that the users’ needs are continuing to be met. A new requirements 
analysis will also reveal requirements that result from advances in Tomahawk 
Land Attack Missile (TEAM) technology. 

• Increase the size of the secondary screen - The extremely small size of the 
secondary screen severely restricts the design of displays designated for this 
screen. A larger screen would increase the possible design alternative available 
for displays such as the ECO displays. 

• Add remote displays to the hardware configuration of ATWCS - The nature of 
the ECO’s responsibilities require that they remain standing during a TEAM 
strike, preventing them from sitting at a stationary console. Remote displays in 
addition to the secondary screen would allow the ECO to keep appraised of a 
strike’s status without having to continually return to the ATWCS consoles. 
Along with this, the ECO could be provided some form of remote control to 
change the display shown on the remote screens. 

C. SUGGESTIONS FOR FURTHER RESEARCH 

Below is a list of areas that present opportunities for further research. 

• Perform heuristic evaluation on the ECO displays with several more evaluators - 
As shown in Nielsen and Molich (1990), the number of usability problems found 
by heuristic evaluation increases significantly when aggregates of evaluators are 
used. These results could be compared to those found by a single evaluator to 
improve the ECO displays and further validate the findings of Nielsen and 
Molich (1990). 

• Perform a summative evaluation of the ECO displays - Quantitative tests will 
uncover fijither usability problems and aid in the validation of the interface 
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design. Measurement tests could also be used to compare the ECO displays 
presented here with alternative designs, 

• Perform a task analysis for the ECO displays - A task analysis would 3deld 
representative task that would need to be accomplished using the ECO displays. 
These tasks could be used to further refine the ECO displays and to perform 
additional usability testing using the cognitive walkthrough method. 
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